The present invention generally relates to image processing technology, and more particularly relates to a technique of processing an image by utilizing information about the three-dimensional position of an object.
FIG. 8 illustrates a basic arrangement for a rangefinder that can capture a range image (or depth image). The rangefinder shown in FIG. 8 pertains to the art closely related to the present invention and is disclosed by us in Japanese Patent Application No. 11-144097, which is hereby incorporated by reference. As shown in FIG. 8, the rangefinder includes camera 51, light sources 52a and 52b, light source controller 55 and distance calculator 56. In response to a vertical sync signal supplied from the camera 51, the light source controller 55 gets each of the light sources 52a and 52b alternately flashed. The distance calculator 56 generates a range image from an image captured by the camera 51.
FIG. 9A is a perspective view illustrating exemplary configurations for the light sources 52a and 52b. As shown in FIG. 9A, flash lamps 57 and 58 such as xenon lamps are vertically stacked in the light sources 52a and 52b, respectively. Reflectors 59 and 60 are placed behind these lamps 57 and 58, respectively, so that the range of the light reflected off from one of these reflectors 59 and 60 horizontally shifts from that of the light reflected off from the other. FIG. 9B is a plan view of the light sources 52a and 52b shown in FIG. 9A. As shown in FIG. 9B, the light sources 52a and 52b radiate, or project, light beams within the ranges A and B, respectively. In the illustrated example, the xenon lamps have so small emissive portions that these lamps can be virtually regarded as point light sources in the plan view. Also, the light sources 52a and 52b are vertically spaced apart from each other by about 1 cm. Accordingly, the light may be regarded as being emitted from almost a point.
Hereinafter, the operating principle of the rangefinder shown in FIG. 8 will be described with reference to FIGS. 10 through 13.
FIG. 10 schematically illustrates exemplary light patterns that have been radiated from the light sources 52a and 52b shown in FIG. 9. In FIG. 10, the solid lines La and Lb represent the brightness on a virtual screen Y, on which the light beams have been projected from the light sources 52a and 52b. The higher the solid lines La and Lb in the direction indicated by the arrow in FIG. 10, the brighter the light projected. As can be seen from FIG. 10, the light projected from each of these light sources 52a and 52b is intensest, or brightest, on the center axis of the projection range and gets gradually weaker, or darker, toward the edges of the range. A distribution like this results from the disposition of the semi-cylindrical reflectors 59 and 60 behind the flash lamps 57 and 58, respectively. And depending on which directions these reflectors 59 and 60 face, the light beams projected from the light sources 52a and 52b may or may not partially overlap with each other.
FIG. 11 is a graph illustrating a relationship between the angle φ of the projected light as measured in the direction H shown in FIG. 10 and the light intensity. In this case, the direction H is defined as a direction in which an arbitrary plane S, including the respective center of the light source and lens, and the virtual screen Y intersect with each other. The angle φ is an angle formed by the light, which has been projected onto the XZ plane, with the X-axis. As shown in FIG. 11, a range α of the light pattern projected from one of the light sources 52a and 52b through the object's space has relatively bright and relatively dark parts on right- and left-hand sides of the light source 52a or 52b, respectively. Conversely, the range α of the light pattern projected from the other light source 52b or 52a through the object's space has relatively dark and relatively bright parts on right- and left-hand sides of the light source 52b or 52a, respectively. It should be noted that the light patterns shown in FIG. 11 change in the height direction (i.e., the Y direction). In other words, the patterns are changeable depending on the level at which the plane including the centers of the light sources and lens is located.
FIG. 12 is a graph illustrating a relationship between the angle φ of the projected light and the light intensity ratio in the range α shown in FIG. 11. As shown in FIG. 12, there is a substantially linear relationship between the light intensity ratio and the angle φ in the range α.
To measure the distance of an object, two types of light patterns should be alternately projected in advance onto a plane standing vertically at a predetermined distance from the light source and light beams reflected from the plane should be received and imaged at the camera 51. A relationship between the light intensity ratio and the angle of the projected light such as that shown in FIG. 12 should be obtained in advance for each Y coordinate (corresponding to a Y coordinate on the CCD). And the light sources 52a and 52b should be disposed so that a line connecting the center of the camera lens to the light sources 52a and 52b is parallel to the X-axis of the CCD imaging plane. In that case, the distance of the object can be estimated accurately based on the data representing the relationships between the light intensity ratios and angles of the projected light associated with the respective Y coordinates obtained beforehand.
Now, take a look at the point P shown in FIG. 8. First, the intensity ratio of the light beams projected from the light sources 52a and 52b onto the point P is obtained from the image captured by the camera 51. And the angle φ corresponding to the point P as viewed from the light sources 52a and 52b can be derived from the resultant intensity ratio by reference to the relationship shown in FIG. 12 associated with the Y coordinate of the point P. Also, the angle θ formed by a visual axis, which extends from the center of the camera 51 to the point P, with the X-axis can be obtained based on the coordinates of a pixel associated with the point P and various camera parameters including coordinates of the optical center of the lens system. Then, the distance of the point P is estimated by the triangulation technique using the two angles obtained θ and φ and a baseline length D, i.e., the distance between the position of the light sources 52a and 52b and the optical center of the camera 51.
Suppose the optical center of the camera 51 is defined as the origin O of the coordinate system; the optical axis of the camera 51 as the Z-axis thereof; horizontal and vertical directions as the X- and Y-axes thereof; an angle formed by a visual axis extending from the light sources 52a and 52b to the point P with the X-axis as φ; an angle formed by a visual axis extending from the camera 51 to the point P with the X-axis as θ; and the coordinates of the light sources 52a and 52b are (0, −D) (i.e., the baseline length is D). Then, the depth Z of the point P is given byZ=D tan θ tan φ/(tan θ−tan φ)Alternatively, all of the three-dimensional coordinates (X, Y, Z) of the point P may be calculated by the following equations using the angle ω shown in FIG. 13:X=Z/tan θY=Z/tan ω
Furthermore, an ordinary color image can also be obtained by adding and averaging the images formed by flashing the light sources 52a and 52b. Accordingly, an image containing three-dimensional (3D) positional information can be captured by using the arrangement shown in FIG. 8.
However, even though it is now technically possible to capture an image with 3D positional information in this manner, the technique per se does not automatically allow each and every maker to market attractive, potentially big-hit products. Accordingly, in developing consumer electronic products, it is very important for the makers to add highly convenient and entertaining functions to the products by taking advantage of the technique.